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Research Summary 

The following research study was conducted during the 2005 - 2006 academic year. Its 
purpose is to help the National Center for Engineering and Technology Education determine 
those engineering outcomes that should be studied in high school when the high school student 
intends to pursue engineering in college. The results of the study will also be used to determine 
those engineering student outcomes that all technology education high school students should 
learn in order to aid them in becoming more technologically literate. 

A modified Delphi approach as used for the study. The participants were a panel of experts 
consisting of engineers, engineering educators, or those expertly fa mi liar with engineering 
education such as a government expert or learned society employee. The modified Delphi study 
ran for six rounds of inquiry during which the panel of experts reached consensus on the identity 
and importance of 43 engineering student outcomes for use in pre-engineering high school 
student learning. The panel of experts also reached consensus on the relative importance of three 
of seven groupings of engineer student outcomes for high school. The results are shown in the 
Summary Table below. 

In the first three rounds of the study, the instrument asked participants to rate outcome 
items on a five point Eikert scale (Clark & Wenig, 1999). The ratings are described below. 

1. Eeast Important: Not necessary for an engineering-related high school curriculum. 

2. Eess Important: Eess than necessary for an engineering-related high school curriculum. 

3. Important: Necessary for inclusion in an engineering-related high school curriculum. 

4. More Important: Essential for inclusion in an engineering-related high school curriculum. 

5. Most Important: Most essential for inclusion in an engineering-related high school 
curriculum. 

After a jury had grouped outcome items into groups of alikeness, the panel of experts was asked 
to rank in the relative importance of the groups. The interquartile range (IQR) was used as the 


statistic for variability of rating and ranking responses (Rojewski & Meers, 1991, Wells, 1994), 



and an IQR of 1 was determined by the researehers to indicate consensus on an item (Wicklein, 


1993). 


Summary Table: Ratings and Rankings of the Outcome Items within Categories 


Rating 

Rank 

Outcome Group and Outcome Consensus Items 

from 

Round 

3 

from Round 
6 



Group Rank 

Group: Eneineerine Desien 

Item 

Ratings 

ist 

Regarding engineering outcomes related to Engineering Design the student in grades 9 through 
12: 

4 


Understands that engineering design is an iterative process. 

4 


Is aware of how engineering principles must be applied when designing engineering solutions to 
problems. 

4 


Understands that creativity is an important characteristic for engineers to apply in design. 

4 


Believes in his/her ability to design a solution to a problem. 

4 


Recognizes that there are many approaches to design and not just one “design process.” 

4 


Understands engineering as it is actually practiced as a future career option. 



Group Rank 

Group: Application of Ensineerins Desisn 

Item 

Ratings 

undetermined 

Regarding engineering outcomes related to Application of Engineering Design the student in 
grades 9 through 12: 

4.5 


Is able to identify problems that could be solved through engineering design. 

4 


Organizes and manages the engineering design process that includes optimal use of materials, 
processes, time, and expertise. 

4 


Designs, produces, and tests prototypes of products. 

4 


Understands that there is no perfect design. Designs that are best in one respect may be inferior in 
other ways (cost or appearance). Usually some features must be sacrificed as trade-offs to gain other 
features. 

3 


Conducts reverse engineering and can analyze how a product or process was designed and created. 

3 


Applies research and development and experimentation in the production of new or improved 
products, processes, and materials. 



Group Rank 

Group: Engineering Analysis 

Item 

Ratings 

3 rd 

Regarding engineering outcomes related to Engineering Analysis the student in grades 9 
through 12: 

4 


Uses models to study processes that cannot be studied directly. 

4 


Applies mathematics and science to the engineering process. 

4 


Uses measuring equipment to gather data for troubleshooting, experimentation, and analysis. 

4 


Understands that knowledge of science and mathematics is critical to engineering. 

3 


Uses a physical or mathematical model to estimate the probability of events. 

3 


Uses optimization techniques to determine optimum solutions to problems. 


IV 


Summary Table Continued 


Group Rank 

Group: Eneineerins and Human Values 

Item 

Ratings 

undetermined 

Regarding engineering outcomes related to Engineering and Human Values the student in 
grades 9 through 12: 

3 


Practices engineering ethics. 

4 


Is aware of how societal interests, economics, ergonomics, and environmental considerations 
influence a solution. 

4 


Understands how other factors, such as cost, safety, appearance, environmental impact, and what 
will happen if the solution fails must be considered when designing engineering solutions to 
problems. 

4 


Takes human values and limitations into account when designing and solving problems. 

4 


Understands that the solution to one problem may create other problems. 



Comment: Understands that engineers have societal obligations and responsibilities. 
(Temporarily added by juror to provide you with a better characterization of this grouping 
of outcomes.) 



Group Rank 

Group: Eneineerine Communication 

Item 

Ratings 

undetermined 

Regarding engineering outcomes related to Engineering Communication the student in grades 
9 through 12: 

4 


Understands basic personal computer operations and uses basic computer applications such as word 
processors, spreadsheets, and presentation software. 

4 


Provides basic technical presentations, graphics, and reports, and communicates verbally information 
related to engineering processes. 

4 


Uses technical drawings to construct or implement an object, structure, or process. 

3.5 


Visualizes in three dimensions. 

3 


Develops and maintains an engineering design portfolio. 

3 


Understands computer-aided engineering. 

3 


Understands scale and proportion in design. 

3 


Applies the rules of dimensioning and tolerancing. 

3 


Uses computer-aided design to construct technical drawings. 



Group Rank 

Group: Engineerine Science 

Item 

Ratings 

undetermined 

Regarding engineering outcomes related to Engineering Science the student in grades 9 
through 12: 

4 


Understands engineering as it is actually practiced as a future career option. 

4 


Develops basic ability to use, manage, and assess technology. 

3 


Applies knowledge of basic ergonomics to the engineering process. 

3 


Develops basic skill in the use of tools for material processes. 

3 


Applies basic power and energy concepts. 

3 


Applies knowledge of the processes for manufacturing products to the engineering process. 

3 


Applies knowledge of material processes to the engineering process. 

3 


Applies knowledge of basic mechanics to the engineering process. 

3 


Applies knowledge of basic statics and strengths of materials to the engineering process. 

3 


Applies knowledge of basic dynamics and motion of rigid bodies and particles to the engineering 
process. 


Summary Table Continued 


Group Rank 

Group: Emereine Fields of Eneineerine 

Item 

Ratings 

^th 

Regarding engineering outcomes related to Emerging Fields of Engineering the student in 
grades 9 through 12: 

3 


Understands the importance of nanotechnologies in developing the next generation of innovations 
(less power, smaller). 

3 


Understands the convergence of nanoscience, biotechnology, information technology and how 
cognitive science creates opportunities for the improvement of industrial productivity and quality of 
human life. 





Abridged Recommendations 

The following recommendations will be of interest to teacher educators, teachers of 

technology education, teachers of pre engineering, engineering educators, administrators, and the 

NCETE. 

1 . One advantage of conducting a Delphi study is that people who may have outstanding stature 
or who may tend to dominate discussions, have less biasing influence on the consensus- 
building process. Nevertheless, it may well be that some decisions are best made in face-to- 
face meetings. Therefore, it is recommended that a series of work sessions be conducted on 
engineering outcomes, in which experts have a chance to more deliberately persuade one 
another about the importance of outcomes and groupings of outcomes. 

2. Conduct a replication study. 

3. Enhance technology education by infusing selected engineering outcomes into the 
technology education curriculum for non-pre engineering curricula, which focus on 
technological literacy. A study to determine which engineering outcomes should be included 
is being conducted with experts in the field of technology education. 

4. Use the outcomes of the study described herein to aid in the design of pre engineering 


programs. 


VI 


5. Use the outcomes of the study described herein to review existing pre engineering programs. 

6. Use the outcomes of the study described herein as contexts to make mathematics and science 
more practical and motivating. 

7. Upon findings in the technology education study recommended above, recommend a listing 
of engineering outcomes that can be infused into technology education programs for the 
purpose of providing technological literacy. 

Implications for Technology Education Curriculum and Instruction 

Some of the implications of the core engineering outcomes identified herein are evident. 
For example, it is clear that engineering education at the K-12 level should be hands-on 
(Douglas, Iverson, & Kavandurg, 2004). So it would be necessary to include outcomes such as 
those related to conducting reverse engineering, research and development, and the fabrication of 
prototypes. It also seems fairly obvious that any program would include a breadth of engineering 
communication activities related to presenting findings, to using CAD, to using the computer as 
a means to control data and communicate engineering processes. Any program that teaches 
engineering would benefit from having students apply mathematics and science principles to the 
solutions that they design. In the midst of an extended back-to-basics movement with high-stakes 
testing, being able to improve student achievement in, and attitudes toward STEM subjects 
would provide a meaningful service to education and, perhaps cause an increase in the diversity 
of those students who would like to pursue STEM related careers after high school and college. 

What engineering outcomes should be included in a high school technology education 
program that focuses on providing students with technological literacy? Certainly, those 
outcomes that most closely correspond to the Standards for Technological Literacy, such as 
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optimization, the realization that there are many societal factors that influence engineered 
solutions, and any outcome that will help students become better designers and understand the 
essence of what engineering is in real life, such as prototyping, creativity, and clearly managing 
the design process. Research and development and analysis are also important. 

What engineering outcomes should be included in a high school technology education 
program that focuses on pre engineering? All of those consensus outcomes from the study 
described herein were identified on the premise that they were to be taught to high school 
students who want to pursue engineering after they graduate. However, the curriculum designer 
should be careful. A crowded curriculum, which leaves no time for application, diminishes its 
effect on student achievement and motivation. Some outcomes need to be taught and applied 
repeatedly across the school year. These fundamental processes are the essence of engineering. 
Other outcomes need only be taught and applied once within a specific course. Perhaps the most 
pertinent approach to deciding what outcomes to include in a pre-engineering curriculum is 
building a course sequence that includes the outcomes in order of importance but also in order of 
prerequisites. Couple with that sequence, estimates of time to deliver instruction in a hands-on 
and motivating way. Where necessary, pre-engineering programs that use these outcomes should 
consider dividing content so it is studied over a sequence of courses over a sequence of grade 
levels, while avoiding too many prerequisite courses that will limit enrollment. 

Having identified those core engineering concepts that should be taught to high school 
students, under what circumstances should one go about teaching the concepts? Douglas, 

Iverson, and Kavandurg (2004) in summarizing the results of an ASEE analysis of current 
practices in K-12 engineering education, developed the following guidelines for the future of K- 
12 engineering education. One, engineering education should be hands-on in order to motivate 
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students by couching engineering problems in interesting and relevant social contexts. Two, 
engineering education should be taught in an interdisciplinary approach in order to show the 
relevancy of mathematics, science, and other subjects, by making engineering a conceptual place 
for the application of these subjects. Three, develop K-12 standards for use in lesson plans that 
help teachers teach mathematics and science concepts in the classroom. Douglas, Iverson, and 
Kavandurg suggest that state-developed K-12 standards should be developed like Massachusetts 
has published. Four, improve teachers by providing more pay, more professional development, 
and more curriculum writing. Five, make engineering a more attractive career choice for girls 
and minorities by working with their schools through outreach efforts. Six, engage more 
constituents in partnerships that cross all levels of the educational process. 

Teachers, teacher educators, and administrators, carry a heavy burden. Their mission is to 
provide students with the best education possible. Traditional education, memorizing facts, and 
studying textbooks has worked to some extent for some professions. However, the world has 
reached a crossroads; a point at which the modus operandi no longer works in terms of fostering 
inclusiveness. The engineering profession and all STEM subjects can attract a more diverse 
population of participants by providing access. This is not simply access to school and books but 
access to instruction that has meaning for students who have not traditionally pursued STEM 
professions. Teaching meaningful concepts and providing meaningful opportunities for 
application is part of what "access" to education is about. 
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Introduction 

In the fall of 2004, the National Center for Engineering and Technology Education 
(NCETE), secured funding from the National Science Eoundation (NSE) in order to fulfill the 
following long-term goals: 

- Preparing graduate students who will become educational leaders engaged in teacher 
preparation and professional development projects with the knowledge and skill to integrate 
engineering into technology education. 

■ Conducting research on how students learn technological concepts, how students learn 
creative problem solving, assessment and evaluation strategies, and how to better prepare 
technology and engineering teachers. 

■ Conducting professional development for grade 9-12 teacher partners based on the testing, 
adaptation, and adoption of instructional techniques that enhance science, technology, 
engineering, and mathematics (STEM). 

- Increase the number and diversity in the pathway of students selecting STEM careers. 

Purpose and Research Questions 

As a basic step in reaching the goals above, the researchers in cooperation with the NCETE 
designed a study to answer the following specific research question: 

Eor grades 9- 12, what should be included in a technology education curriculum that 
infuses engineering design, where the goal of the curriculum is technological literacy? 
However, as a prerequisite to that question, the Center needed to determine what engineers 
believe students should learn in high school. To frame that prerequisite part of the study, the 


researchers posed the following preliminary research question, which is the focus of this article: 
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What are the engineering student outcomes that prospective engineering students in grades 
9-12 should know and be able to do prior to entry into a post-secondary engineering 
program? 

For the purpose of answering this prerequisite question, statements of outcomes of student 
achievement were sought through a modified Delphi study. 

The following review of related literature provides the reader with an understanding of 
where the outcomes provided to the panel of experts participating in the present study originate, 
and it also provides the reader with important background knowledge to which he or she can 
compare the findings of the present study. 

Related Literature 

As discussed above, the researchers want to ultimately identify those engineering outcomes 
that will help technology education teachers develop technological literacy in their students. 
Therefore, the following few sections briefly address technological literacy. 

What is technology? “Technology is the modification of the natural environment in order 
to satisfy perceived human wants and needs” (International Technology Education Association, 
2000, p. 9). There are many other definitions of technology in the literature. The one above may 
not exactly match the one that the reader accepts, but the point of the definition is that it is 
broader than the general public perceives technology. Technology is more than just computers 
and electronics. It includes a wide variety of concepts, constructs, and activities, including 
engineering. 

What is technology education? Technology education is the school subject that has as its 
purpose the development of students’ technological literacy. All students in grades K - 12 should 
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enroll in technology education classes, whether they want to pursue technology related careers or 
not. This is because technological literacy is a characteristic that all citizens should possess 
because they live in a society that is influenced in every way by technology (Dyrenfurth, 1991). 

What is technological literacy and why is it important. “Technological literacy is the 
ability to use, manage, assess, and understand technology” (International Technology Education 
Association, 2000, p. 9). The idea that technological literacy is important for every student in 
school relates back to the idea that technology influences every aspect of society. Therefore, any 
particular student who enrolls in technology education may become a business person, 
homemaker, engineer, teacher, clergyperson, grocery clerk, or any other manner of productive 
citizen once he or she graduates and enters adulthood. Furthermore, technological literacy may 
be thought of as existing within different people along a continuum of sophistication. For 
example, a high school graduate may have some technological ability and knowledge that serves 
to help solve everyday problems that he or she may encounter. At the other end of the 
continuum, an engineer may possess a higher level of technological literacy based on his or her 
education and extensive experience (Dyrenfurth, 1991). However, both of these people, to some 
extent, use their technological literacy in order to improve their daily lives. Bordogna (1997) 
expressed similar views when he wrote: 

To be personally successful in today's world and simultaneously promote prosperity, 
engineers need more than first-rate technical and scientific skills. In an increasingly 
competitive world, engineers need to make the right decisions about how enormous 
amounts of time, money, and people are tasked to a common end. I like to think of the 
engineer as someone who not only knows how to do things right but also knows the right 



13 


thing to do. This requires engineers to have a broad, holistie baekground. Sinee engineering 

itself is an integrative process, engineering education must focus on this end (n.p.). 

It seems the profession of engineering is trying to develop a more broad perspective on the 
nature of engineering and the role of broad goals in engineering education. In describing the 
setting in which engineers will work in the year 2020, the National Academy of Engineering 
(2004) simultaneously describes the technological society in which all citizens will live. 

• The population of individuals who are involved with or affected by technology (e.g., 
designers, manufacturers, distributors, users) will be increasingly diverse and 
multidisciplinary . 

• Social, cultural, political, and economic forces will continue to shape and affect the 
success of technological innovation. 

• The presence of technology in our everyday lives will be seamless, transparent, and 
more significant than ever. (p. 53) 

In the sense that technological literacy is needed by all citizens, the rationale for 
technological literacy is not only an economic one. In Technically Speaking..., Pearson and 
Young (National Academy of Engineering, 2002) make a strong case for “technical literacy” 
better insuring the economic well being of the United States. However, while the rationale for 
technological literacy is certainly economic, in part, it is developed to benefit all citizens. 

Another way to think of technological literacy is that it is as fundamental as any citizen 
having basic literacy; being able to read well, write well, and speak well is fundamental. 
Technological literacy is equally fundamental. Being able to use, manage, assess, and understand 
technology, provides some level of technological literacy; a key ability for succeeding in a 
technological society. There is a definite relationship between technological literacy and 
engineering in the PK-12 span of engineering education. 
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Infusing Engineering Design Processes into the Technology Education Curriculum 

Emphasis in engineering within the technology education curriculum is not a new idea. 
Olson (1957) suggested the inclusion of engineering concepts in industrial arts education in the 
late 1950s. Lewis (2004) summarizes the breadth of the effort to integrate engineering into the 
technology education curriculum. While the Massachusetts Department of Education (2001) has 
developed an extensive set of content standards for its own pre-engineering curriculum (as have 
many other states in the US), Lewis documents that a variety of states are allowing students to 
take Project Lead The Way courses, a pre-engineering approach, as part of their technology 
education. However, Lewis also characterizes the pre-engineering emphasis as both a way to 
integrate STEM education thus improving student achievement and as a way of improving the 
perception of technology education among educators and other professionals from other 
academic disciplines. 

In order to improve the level of acceptance that technology education can gain in the public 
schools and in order to better represent the essence of engineering as it relates to technology for 
the improved achievement of students, Wicklein (2006) proposes infusing engineering design 
into the technology education curriculum more deliberately. He outlines broad categories for the 
infusion of engineering design into technology education. In terms of those broad areas of 
engineering that should be infused into the curriculum he includes, “. . .narrative descriptions, 
graphical explanations, analytical calculations, physical creation” (p. 7). He also describes 
courses that might represent a technology education curriculum that infuses engineering design. 
The courses include, “Introduction to Technology, Engineering Graphics, Research and Design, 
Engineering Applications” (p. 6). He includes as essential in the curriculum optimization. 
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analysis, and prediction. Wicklein also implies that students should take all of the science and 
mathematics courses that are available in high school. 

Existing Efforts to Identify and Integrate K-I2 Engineering Concepts 

Lewis (2004) has also done a comprehensive job of summarizing efforts within 
technology education to integrate the curriculum with science, engineering, and mathematics. 
Projects such as the Integrated Mathematics, Science, and Technology Project (Satchwell & 
Loepp, 2002) and the Technology, Science, Mathematics Integration Project (Childress, LaPorte, 
& Sanders, 1994) are just two of several efforts to integrate STEM education that were headed 
by technology education professionals. However, there are also efforts outside of the field of 
technology education. Programs such as those in the Centers for Learning and Teaching (2005), 
supported by the National Science Foundation (NSF), are attempting, in some form, to integrate 
STEM education at the public school level. NSF funding has also included money for informal 
STEM education targeted at the K-12 and family levels. The Boston Museum of Science (2005) 
is one example of such outreach efforts. 

McREL 

Mid-Continent Research for Education and Learning (McREL) (2004) is an example of a 
U.S. Department of Education effort to provide standards for the integration of STEM and other 
school subjects. McREL is charged with creating reform in education through systemic 
initiatives, and its fourth edition of a compilation of school- wide content standards provides, 
perhaps, one of the most comprehensive sets of standards available to teachers. McREL and the 
Benchmarks for Science Literacy are the two best, easily accessible resources to find core 
engineering concepts that should be taught at the high school level in terms of breadth of 
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coverage. The engineering section for McREL is substantial, and below is one example of core 
engineering concepts identified by McREL. 

“Standard 14. Uses the design process to solve problems 

Understands that engineering design is an iterative process involving modeling and optimization 
to find the best solution within given constraints” (2004, N.P.). 

Curriculum and Evaluation Standards for School Mathematics 

The first notable set of national standards was developed in the late 1980s by the National 
Council of Teachers of Mathematics. It formed the Commission on Standards for School 
Mathematics, which developed broad standards for mathematics education in the public schools. 
The standards are grouped into large categories, and a great emphasis is placed on developing the 
student as a problem solver as opposed to one who memorizes mathematical facts. In both the 
original and revised versions, the Commission’s publication. Curriculum and Evaluation 
Standards for School Mathematics, emphasizes that “less is more” when it comes to freeing up 
enough time in the classroom to develop students who use mathematics reasoning and problem 
solving. A very refreshing feature of these standards is that an effort is made to emphasize the 
use of mathematics in other subject areas such as science and technology (Commission on 
Standards for School Mathematics, 2000). 

Science Standards and Engineering 

Beginning in the late 1980s and through the 1990s three notable sets of science education 
standards were developed. Two emphasized the importance of teaching technology and 
engineering in the science curriculum. The three projects are briefly described below. 

• Scope, Sequence, and Coordination of Secondary School Science, developed by the National 
Science Teachers Association (1992), did not directly call for the integration of science and 
technology, but it laid a foundation for later work in science content standards. 
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• Science for All Americans (1989) and the Benchmarks for Science Literacy (1993) developed 
by Project 2061 of the American Association for the Advancement of Science, called directly 
for curriculum integration of mathematics, science, and technology. 

• National Science Education Standards, developed by the National Research Council (1996), 
also included standards that related to technology and engineering. 

However, among the science standards projects, the most explicit statements of what 
students should know and be able to do related to interfaces among STEM subjects, and those 
especially related to engineering and technology are identified in the Benchmarks for Science 
Literacy. The Benchmarks... provides the most well phrased items available regarding core 
engineering concepts for high school students and is worth a closer examination. In the context 
of the more broadly learned engineer described by Bordogna (1997) and the integration of STEM 
content described by Salinger (2003), the Benchmarks (AAAS, 1993) describes the interaction of 
technology and science such that students should leave school with the understanding that 
technological innovation is often enhanced by science knowledge and processes of inquiry. The 
Benchmarks describes the interaction and interdependence of technology and society including 
detailed statements about the economy, government regulations, and human needs. The Designed 
World is a set of standards related to a variety of specific technologies such as agriculture, 
medicine, communication, and manufacturing. Benchmarks for Science Literacy even has a 
section on mathematics, statistical analysis, uncertainty, and mathematical symbolism. The group 
of standards most closely related to engineering and engineering design is included in a section 
called "Design and Systems." Design and Systems standards provide some of the core 
engineering concepts that would need to be included in a high school level engineering design 
course. The following is one example of the pertinent benchmarks. 
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“Design usually requires taking constraints into account. Some constraints, such as gravity or the 
properties of the materials to be used, are unavoidable. Other constraints, including economic, 
political, social, ethical, and aesthetic ones, limit choices” (N.P.). 

Standards for Technological Literacy 

In 1996, the International Technology Education Association (ITEA), with funding from 
the NSE and the National Aeronautics and Space Administration began the Technology for All 
Americans Project, which culminated in 20 standards, and their benchmarks, for technology 
education and other programs that contribute toward developing technological literacy in public 
school students. In 2000, ITEA published the Standards for Technological Literacy: Content for 
the Study of Technology. In addition to helping teachers develop curricula related to technology 
as it is broadly defined, these standards and their benchmarks call for students to understand a 
number of concepts related to engineering, including optimization, trade-offs, engineering 
design, and design skills and knowledge. The following is one of the benchmarks related to the 
engineering design standard. 

"Engineering design is influenced by personal characteristics such as creativity... and the ability 
to visualize..." (p. 104). 

Massachusetts Standards for Engineering Design 

The Massachusetts' engineering standards are not as extensive as those developed by 
McREE or the Benchmarks for Science Literacy insofar as they cover pure engineering design 
only to a limited extent. They are more closely aligned with the Standards for Technological 
Eiteracy and industrial technology. Below is one example of the engineering design standards 


cited by Massachusetts. 
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“Identify and explain the steps of the engineering design process, i.e., identify the problem, 
research the problem, develop possible solutions, select the best possible solution(s), construct a 
prototype, test and evaluate, communicate the solution(s), and redesign” (N.P.). 

The Bearing and Daugherty Modified Delphi Study 

Bearing and Daugherty (2004) describe a modified Delphi study that they conducted with 
technology teachers, technology teacher educators, and engineering educators participating. The 
purpose of the study was to identify those concepts that are necessary to teach high school 
students in order to prepare them for postsecondary engineering education, while preserving the 
mission of teaching technological literacy. Bearing and Daugherty developed a predetermined 
list of concepts based on information from Project Lead The Way, Principles of Technology, the 
Standards for Technological Literacy, American Society of Engineering Education, and others. 
Participants were to decide if a concept should be included in a curriculum or not included in a 
curriculum. Eifty-two concepts on their list met the criterion for consensus and were retained. 
Items were then ranked in order of importance. Below is the top concept in their study. 
“Interpersonal Skills: teamwork, group skills, attitude, work ethic” (p. 10). 
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Australian Ministry of Education 

The Curriculum Council (Australian Ministry of Education, 2004) has created a very 
sophisticated set of outcomes for achievement in the high school grades regarding engineering. 
These outcomes are organized into a course called Engineering Studies. The course has four 
overarching engineering outcomes and many specific or prerequisite outcomes organized under 
the overarching ones. The first of these engineering outcomes is: 

“Engineering Process: Students apply a process to design, make, communicate ideas and 
evaluate components” (p. 8). 

There has been a progression of events leading up to the study described herein. The desire 
to improve student achievement is chief in the motivation to infuse engineering design processes 
into the technology education curriculum. Improving the perception of technology education is 
an important part of the motivation to integrate technology and engineering at the 9-12 level. The 
study described herein, seeks to build on the foundation that has already been laid by the 
aforementioned national standards projects and identify outcomes for student achievement in 
high school engineering education and later in technology education programs by infusing 
engineering design into the technology education curriculum. 

Additional Resources Identifying Engineering Concepts 

One might wonder why not just pattern high school engineering after university 
engineering programs. However, a curriculum designer or committee should not simply start 
reviewing ABET standards (2005) with the hope of identifying what should be taught in high 
school (although several ABET standards are appropriate). The Dearing and Daugherty (2004) 
modified Delphi study and the one described near the end of this article underscore the reality 
that engineers and technology educators do not necessarily believe that what should be taught at 
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the high school level should be the same as that which is taught at the university level. In 
addition to determining what is at the core of engineering concepts, it is also necessary to 
identify emerging engineering concepts. What are those things about the engineer of the future 
that high school students should know beyond the core. One could look to publications like The 
Engineer of 2020 (National Academy of Engineering, 2004) and see what committees of leading 
engineers believe about the future of engineering. There are other resources for identifying 
engineering concepts to be taught in the K-12 schools (see also Koehler, Faraclas, Sanchez, 

Latif, and Kazarounian, 2005). The authors of this article have not only identified those 
resources described above, but they have also located some 15 additional state-related or project- 
related sources of outcomes, all of which are either similar to or not as useful for core 
engineering to those described above. 

Characterizing Engineering Knowledge and Skills 

Two more resources for identifying engineering concepts remain for discussion prior to 
describing the authors' study on engineering outcomes. These are two resources (in addition to 
the Bearing and Daugherty study) that will help the curriculum developer have confidence in 
what to include in a high school engineering program. Identifying core concepts in engineering is 
not necessarily a simple task. Researchers have many purposes when it comes to identifying 
those concepts and skills that engineers need to know and be able to do. Perhaps to some extent 
those things that are expected of practicing engineers should be reflected in the high school 
engineering curriculum. Robinson, Sparrow, Clegg, and Birdi (2005) surveyed and interviewed 
58 design engineers regarding the future importance of competencies needed by design engineers 
in the coming decade. Their purpose was to inform design engineering firms how to remain 
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competitive. However, everything that an engineer needs to know and be able to do is not 
necessarily something that characterizes engineering and allows one to differentiate engineering 
from other technical fields. In fact, it appears that many of the emerging concepts and skills that 
engineers will need in the future are the same concepts and skills needed by people in non- 
engineering fields. Robinson, Sparrow, Clegg, and Birdi also identified core competencies, and 
they defined them as those competencies which do not change over time and which retain their 
importance. 

Emerging Engineering Concepts 

There has been a long tradition of engineering in modern society. The process of 
engineering design has been associated with many significant milestones of technological 
advancement. Some advancement is revolutionary, and some of this advancement is incremental. 
Robinson, Sparrow, Clegg, and Birdi state, for example, that the invention of the jet engine was a 
revolutionary advancement in powered flight. However, since the invention of the jet engine, 
advancement in jet engine technology has been incremental. They believe that in the future there 
is going to be more incremental technological advancement than there will be revolutionary 
technological advancement. 

Innovation and creativity are two of the core competencies that Robinson, Sparrow, 
Clegg, and Birdi (2005) identify, and they discuss their relative importance in the future. They 
assert that creativity is more important to the process of designing revolutionary inventions. 

Since revolutionary invention occurs less often, they reason that incremental innovation is going 
to be a more important competency overall compared to creativity. As a company struggles to 
gain an edge over its competition, being able to sustain incremental innovation is not only 
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possible, but also desirable. On the other hand, creativity is associated with large breakthroughs 
or revolutionary improvements, and in a relative sense, creativity is more difficult to sustain at a 
level that will make a difference in a highly competitive economy. Because of the changing 
nature of competition in the global economy, incremental innovation will emerge as an important 
competency in the future they contend. This seemingly contradicts what the National Academy 
of Engineering (NAE) (2004) postulates for engineering attributes needed for the year 2020. It 
maintains that creativity is not only an important core engineering concept now, but it will 
become even more important as the global economy strengthens and the rate of technological 
innovation skyrockets in the future. 

While innovation and creativity are widely debated and are considered important to 
engineering, Robinson, Sparrow, Clegg, and Birdi did not find them to be the most important 
competencies for the coming decade. They found that non-technical skills will emerge in the 
future as being so important as to rival technical skills. These competencies include skills like 
managing projects well and being open minded. In the future, the design engineer will have to 
lead groups that are comprised of both engineers and non-engineers. Robinson, Sparrow, Clegg, 
and Birdi (2005) found the following as the ten most important engineering competencies. 

1. plans work, 6. is knowledgeable about engineering 

2. judges importance 7. is motivated or works hard 

3. makes effective decisions 8. analyzes tasks 

4. monitors progress 9. thinks intuitively 

5. uses effective learning methods 10. uses appropriate communication formats 

These findings tend to support what Bordogna (1997) emphasizes about the holistic 
engineer. It is also reassuring that, with the exception of creativity, the findings of the Robinson, 
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Sparrow, Clegg, and Bird! study reinforce the opinions expressed by the NAE in The Engineer of 
2020 and reinforce the findings of the Bearing and Daugherty (2004) modified Delphi study. In 
deed, the engineer of 2020 will be solving problems in a world with high volume data and 
information flow, an accelerating global economy, diversifying global demographics, and more 
demanding customers in the face of increased competition. The engineer of 2020 will need the 
following emerging attributes: high ethical standards and professionalism, flexibility, dedication 
to lifelong learning, and teamwork. The engineer of 2020 will need to understand nano, bio, 
optical, and smart materials technologies. The engineer of 2020 will better understand the needs 
of society and understand government, politics, economics, business, and leadership, and be a 
good communicator (NEA, 2004). What is implied by both Robinson, Sparrow, Clegg, and Birdi 
and the NAE is that emerging competencies and attributes will become core engineering 
competencies and attributes. In fact, the NAE suggests that a fifth or sixth year will need to be 
added to the traditional undergraduate engineering curriculum in order to accommodate these 
additional attributes, which the NAE considers to be essential. 

Core Engineering Concepts 

Robinson, Sparrow, Clegg, and Birdi (2005) define "core competency" as a competency 
that is currently essential and which will remain essential. If core competencies are not 
necessarily those things that are unique to engineering, what are those unique competencies? 
Clearly Wicklein (2006) believes that engineers more than other designers, apply mathematics 
and science to the design of technological solutions to problems in deliberate, efficient ways. 
Because engineers seek to be efficient, engineering design has evolved into a relatively 
constrained process. Wicklein postulates that optimization, analysis, and prediction are the three 
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things that separate engineering design and other forms of design. Optimization is the use of 
mathematics and science in order to create the most reliable designs. This is reinforced by the 
NAE that identifies the following core attributes for engineers in the year 2020. Engineers will 
continue to need strong analytical skills and be able to apply principles of mathematics and 
science to the design process. Engineers will continue to be good design project planners who 
can structure a project and run it efficiently in order to achieve a desired outcome. The findings 
of Robinson, Sparrow, Clegg, and Birdi (2005) tend to support what the NAE writes about core 
engineering attributes. Currently, creativity and innovation are also considered core attributes by 
the NAE. 

This review of literature has provided the reader with an understanding of where the 
outcomes provided to the panel of experts participating in the present study originate, and it also 
provides a foundation upon which the reader may compare the findings of two highly related 
writings to the present study. Those related writings are the NAE’s Engineer of 2020 and the 
findings of Robinson, Sparrow, Clegg, and Birdi. Next, the current researchers present their 
research study. 


Methodology 

Phase I: Focus Groups 

The researchers conducted two focus groups in order to gather input on what engineering 
concepts should be taught at the high school level. The first focus group consisted of two 
technology teacher educators, one supervisor, and one engineer. The second focus group 
consisted of three technology education teachers and one mathematics teacher. The criteria for 
selection on the focus groups were the participant needed to be: 
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• knowledgeable about engineering and technology education, and 

• Well recommended by a peer regarding his or her engineering knowledge. 

Participants were given informed consent and were paid $70 for participation. 

In conducting the focus groups, the researchers had a guide sheet which was used to keep the 
group moving from one topic of interest to the next. The researchers were careful to avoid 
interrupting unnecessarily and to avoid influencing opinions of the participants. High quality 
audio equipment was used to record sessions and transcripts were keyed and printed. The 
researchers analyzed the audio recordings for themes and trends. The researchers also wanted to 
conduct two focus groups composed of engineers only, however, the researchers were 
unsuccessful in recruiting a sufficient group of engineers to agree to gather in one location. After 
a significant amount of time passed, the researchers decided to scrap the idea of conducting focus 
groups with prominent engineers or any engineers all together. Based on their review of the 
literature and the findings of the first two focus groups, the researchers decided to begin the 
modified Delphi study. 

Phase II: Modified Delphi Study 

This second phase of the outcomes study used a modified Delphi approach that started with 
preexisting outcome items selected from national standards projects, the phase one focus groups, 
and additional resources. The modified Delphi study extended for three rounds with 34 
participants as of Round 2 and 32 participants as of Round 3 (Dalkey, 1972; Custer, Scarcella, & 


Stewart, 1999). 
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Identification of Pre Selected Outcomes 

The researchers chose engineering outcomes from the following standards resources: 

• Findings of the focus groups (conducted in fall, 2005 by the researchers) 

• American Association for the Advancement of Science (1993) 

• Mid-Continent Research for Education and Learning (2004) 

• National Research Council (1996) 

• International Technology Education Association (2000) 

• Massachusetts Department of Education (2001) 

• Dearing and Daugherty (2004) 

• National Council of Teachers of Mathematics (2000) 

• Koehler, Earaclas, Sanchez, Latif, and Kazarounian (2005) 

• Bordogna (1997) 

Eor the most part, standards were taken with the exact same wording as the standards are 
listed by the respective resources above. Sources were not revealed to Delphi participants in 
order to avoid biasing participants’ opinions. However, some wordings were later changed. 

Two engineers and two technology teacher educators, one an accreditation expert and the 
other a former engineer, reviewed the original list of outcomes and reviewed the instructions and 
layout of the Round 1 instrument. They reviewed the instrument to make sure that the outcome 
items were engineering oriented, and that they belonged ontologically and epistemologically. 

The reviewers suggested some rewordings and some changes to the directions. The Round 1 
instrument had 47 outcome items and room for participants to add all of those items that they 
believed should be added. 


Identification of Participants 

The criteria for selection as a participant in the study were that the participant: 

• Is a practicing engineer, engineering educator, or 

• Is working in a field closely related to engineering or engineering education such as a 
curriculum writer or an association/non-profit or government employee, and 
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• Has been professionally active in an engineering organization, or 

• Has had an interest in K-12 education. 

Participants were nominated by a prominent employee of the National Research Council 
and by a former employee of the Accreditation Board for Engineering and Technology. Some 
participants were, in turn, nominated by these first nominees. Approximately 45 participants 
were solicited for participation, and 34 accepted the invitation to participate. They were provided 
with informed consent and were paid $50 for participation in the first three rounds of the 
modified Delphi study. They were paid an additional $25 for participation in subsequent rounds. 

Findings 

Focus Groups 

It is important to note that the focus groups turned out to be effective at identifying issues related 
to infusing engineering content into the technology education curriculum. They were fairly 
successful at yielding content. However, only an abridged list of focus group outcomes findings 
is presented in Table 1. 


Table 1. Abridged summary of focus group findings as they related to engineering 
outcomes. 

• It is important to determine how we define engineering 

• Re engineer and re design things that exist to develop problem solving skills and conceptual skill 

• Engineering disciplines mechanical, civil, electrical, computer engineering, biomedical 

• Thermodynamics, Statics, Design concepts 

• Wbat separates engineering design from general design is the actual process of applying standards 
and functionality to what you are doing 

• How math and science play a part in the field. Equations are used for simulations and used to design 

• Simulations are not necessarily computer simulations but calculating with your calculator 

• Math is the language of engineering. They call it the symbolic language of engineering 

• Introduce them to all the areas of engineering. 

• We have structured classes parallel to statics and classes geared toward dynamics and so forth. 
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• We offer for engineering classes which are construction, transportation, manufacturing and then I 
offer power and energy and they also offer communications. 

• Another teacher says they are going to change to design, research and design and communications. 

• Having them to figure out or make a prediction of how they will use something for performance and 
then the actual physical test of that object is to basically validate their calculations. 


Modified Delphi Study 

Given the general lack of diversity in STEM fields, the researchers were not disappointed 
with the demographic characteristics of the modified Delphi study participants. Table 2 provides 
a summary which includes some indication of the extent to which the participants were qualified 
to participate in the study. 


Table 2. Participant demographics. 


N = 34 participants 


Gender 

Female: 

n=13 

Male: 

n=21 




38% 


62% 



Race* 

Caucasian: 

n=26 

African 

n=4 

Native 

n=l 

Asian: 

n=l 

Mixed: 

n=l 



76% 

American: 

12% 

American: 

3% 


3% 


3% 


Age* 

Mean: 


Range: 


50.67 


71-33=38 


Years of Experience as Engineer* 

Mean: 

Range: 


26 participants are or have 


12.6 

55-0=55 


been practicing engineers 


Years of Experience as Engineering 

Mean: 

Range: 


28 are or have been 

Educator* 

14.18 

40-0=40 


engineering educators 


Years of Experience in Engineering 

Mean: 

Range: 


5 are in jobs related to 

Related Position* 

2.67 

26-0=26 


engineering with a mean: 17.2 


* 1 participant did not respond to the demographic part of the instrument. 


As seen in Table 3, most participants had responsibilities that one would expect of 


professionals in engineering or related to engineering. For example, even though they are 
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working at the four-year college level, three professors are responsible for K-12 outreach. Other 
participants are professors of engineering, administrators, or are responsible for helping write K- 
12 curriculum or help to administer governmental agencies or non-profit organizations. 


Table 3. Current responsibilities. 


Responsibilities Current Position 

Frequency 

Percent 

Valid Percent 

Cumulative 

Percent 

Valid missing 

3 

OO 

bo 

OO 

bo 

OO 

bo 

admin 

6 

17.6 

17.6 

26.5 

assoc dean eng 

1 

2.9 

2.9 

29.4 

dean of eng 

1 

2.9 

2.9 

32.4 

dept head 

1 

2.9 

2.9 

35.3 

design 

1 

2.9 

2.9 

38.2 

dir of center 

1 

2.9 

2.9 

41.2 

dir of curr preeng 

1 

2.9 

2.9 

44.1 

dir of prog 

1 

2.9 

2.9 

47.1 

eng admin 

1 

2.9 

2.9 

50.0 

kl2 coord 

3 

CO 

cb 

CO 

cb 

58.8 

kl2eng edu coord 

1 

2.9 

2.9 

61.8 

teach eng 

8 

23.5 

23.5 

85.3 

teach eng & kl2 curr 

1 

2.9 

2.9 

88.2 

teach eng write curr 

2 

5.9 

5.9 

94.1 

teach manf eng 

1 

2.9 

2.9 

97.1 

teach math and physi 

1 

2.9 

2.9 

100.0 

Total 

34 

100.0 

100.0 



Participant distribution in terms of the type of organization at which the participant is 
employed is also not unusual. Most are university professors with various responsibilities as 
discussed above. However, two participants are currently practicing engineers, three work for 


non-profits, and one is employed by the government. 
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Table 4. Organization type. 


Organization 

Frequency 

Percent 

Valid Percent 

Cumulative 

Percent 

Valid university 

21 

61.8 

65.6 

65.6 

community college 

5 

14.7 

15.6 

81.3 

government agency 

1 

2.9 

3.1 

84.4 

engineering firm 

2 

5.9 

6.3 

90.6 

non-profit 

3 

8.8 

9.4 

100.0 

Total 

32 

94.1 

100.0 


Missing System 

2 

5.9 



Total 

34 

100.0 




In order to keep track of the characteristics of the participants, they were asked to identify 
the engineering discipline in which they were educated. Five participants are not engineers, and 
one participant did not respond, thus the frequency of six at the top of Table 5. 


Table 5. Engineering discipline. 


Engineering Discipline 

Frequency 

Percent 

Valid non-engineer 

5 

14.7 

biomedical 

1 

2.9 

chemical 

1 

2.9 

civil 

2 

5.9 

electrical 

10 

29.4 

electrical and mecha 

1 

2.9 

electronic and mecha 

1 

2.9 

industrial 

1 

2.9 

manufacturing 

1 

2.9 

materials 

1 

2.9 

mechanical 

8 

23.5 

metalurgical 

1 

2.9 

missing 

1 

2.9 

Total 

34 

100.0 


Round 1, Engineering Outcome Ratings 

In Round 1, for engineering outcomes for grades 9-12 for students who want to pursue 
engineering after graduation, participants were asked to rate items, reword items if needed, add 


new items and rate any new items that they added, and provide comments. An explanation of the 
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rating scale is provided in Table 6. Many of the outcome items were very long. Therefore, they 
are abbreviated below in Table 7. 


Table 6. Explanation of ratings. 

The instrument asked participants to rate outcome items on a five point Likert scale (Clark & 
Wenig, 1999). The ratings are described below. 

6. Least Important: Not necessary for an engineering-related high school curriculum. 

7. Less Important: Less than necessary for an engineering-related high school curriculum. 

8. Important: Necessary for inclusion in an engineering-related high school curriculum. 

9. More Important: Essential for inclusion in an engineering-related high school curriculum. 

10. Most Important: Most essential for inclusion in an engineering-related high school 
curriculum. 

The interquartile range (IQR) was used as the statistic for variability of rating responses 
(Rojewski & Meers, 1991, Wells, 1994), and an IQR of I was determined by the researchers to 
indicate consensus on an item (Wicklein, 1993). Because in the beginning, the researchers were 
attempting to group items by their ratings, the median was used to represent the rating that most 
closely characterizes the importanee of the item along with the mean to account for any 
clustering of data at both extremes of the rating scale and due to low a participant pool. Twenty 
items achieved eonsensus after Round 1. Each of these items’ median ratings was either 3 or 4. 


Table 7. Round 1 analysis. 


Item 

Outcome 

IQR 

Mdn 

Mn 

SDt 

1 

Is able to define engineering. 

2 

4 

4.03 

1.19 

2 

Understands engineering as a future career option. 

1.25 

4.5 

4.18 

.999 

3 

Understands the various disciplines of engineering 

2.25 

3 

3.44 

1.24 

4 

Develops basic ability to use, manage, and assess 
technology. 

1.5 

4 

3.91 

.805 

5 

Practices engineering ethics. 

1.25 

4 

3.56 

1.16 

6 

Works effectively in teams which also include non- 
engineers. 

2 

4 

3.82 

1.03 

7 

Understands that engineering design involves identifying 
needs for... 

1.25 

4 

4.18 

.869 

8* 

Uses models to study processes that cannot be studied 
directly. 

1* 

4 

3.59 

.988 

9 

. . . engineering design 

is . . . iterative . . . modeling . . . optimization 

1.25 

5 

4.26 

.898 

10* 

Organizes and manages the engineering design 

1* 

4 

3.62 

.853 
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process. . .optimal use of... 





11 

. . .economics, ergonomics, and environmental. . .influence a 
solution. 

2 

4 

3.79 

.880 

12 

. . .engineering principles . . . applied in designing 
engineering solutions 

1.25 

4 

3.65 

1.10 

13* 

. . . other factors . . . must be considered in designing 
engineering solutions 

1 * 

3.5 

3.68 

.843 

14* 

Uses optimization techniques to determine optimum 
solutions... 

1 * 

3 

2.59 

.857 

15* 

Applies mathematics and science to the engineering process. 

1 * 

4 

4.26 

.898 

16 

Uses a physical or mathematical model to 
estimate. . .probability of events. 

2 

3 

2.94 

1.21 

17* 

. . .reverse engineering. . .can analyze how a product or 
process is designed 

1 * 

3.5 

3.35 

.917 

18 

. . .engineering design includes. . . improvement of existing 
products... 

2 

4 

3.94 

.814 

19* 

. . .creativity is . . .important characteristic for engineers to 
apply in design. 

1 * 

4.5 

4.26 

.864 

20 

Applies research and development and 
experimentation . . . new . . . products 

1.25 

3 

3.26 

1.14 

22 

Designs, produces, and tests prototypes of products. 

1.25 

4 

3.59 

1.08 

23 

... no perfect design. Designs that are best. . .may be inferior 
other ways... 

2 

4 

3.85 

.989 

24* 

Takes human values and limitations into account when 
designing... 

1 * 

4 

3.74 

.864 

25 

Understands that the solution to one problem may create 
other problems. 

2 

4 

3.97 

.883 

26 

Design... requires taking constraints into account. Some are 
unavoidable. 

1.25 

4 

3.85 

.857 

27 

Uses graphs to show a variety of possible relationships 
betwn variables. 

1.25 

4 

4.12 

.913 

28* 

. . .personal computer operations and uses basic computer 
applications... 

1* 

4 

4.18 

.904 

29* 

. . .basic technical presentations, graphics, and reports, and 
commun... 

1* 

4 

4.24 

.890 

30 

Develops and maintains an engineering design portfolio. 

2 

3 

3.18 

1.22 

31* 

Uses technical drawings to construct... object, structure, or 
process. 

1* 

4 

3.64 

1.03 

32 

Understands computer-aided engineering. 

2 

3 

3.00 

.985 

33* 

Understands applications of scale and proportion in design. 

1* 

3 

3.56 

.824 

34* 

Visualizes in three dimensions. 

1* 

4 

3.68 

.976 

35 

Uses technical sketching to communicate in the design 
process. 

1.25 

3.5 

3.53 

1.09 

36* 

Applies the rules of dimensioning and tolerancing. 

1* 

3 

2.76 

1.08 

37 

Uses computer-aided design to construct technical drawings. 

1.25 

3 

2.71 

.970 
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38* 

Applies knowledge of basic ergonomics to the engineering 
process. 

1* 

3 

2.65 

.734 

39 

Applies basic electronics concepts to the engineering 
process. 

2 

3 

3.06 

.983 

40* 

Uses measuring equipment to gather data for 
troubleshooting. . .analysis. 

1* 

4 

4.18 

.716 

41* 

Develops basic skill in the use of tools for material 
processes. 

1* 

3 

3.32 

.843 

42 

Applies basic power and energy concepts. 

1.25 

3.5 

3.53 

1.08 

43* 

Applies knowledge of the processes for 
manufacturing . . . engineering . . . 

.25* 

3 

2.97 

.797 

44 

Applies knowledge of material processes to the engineering 
process. 

1.25 

3 

3.03 

.797 

45* 

Applies knowledge of basic mechanics to the engineering 
process. 

1* 

3 

3.35 

.774 

46 

Applies knowledge of basic statics and 
strengths. . .engineering process. 

2 

3 

3.09 

.933 

47* 

Applies knowledge of basic dynamics and 
motion. . .engineering process. 

1* 

3 

2.71 

1.06 

Added48 

Is able to identify problems that. . .be solved through 
engineering design. 

“ 

“ 

“ 

■ 

Added49 

Believes in his/her ability to design a solution to a problem. 

- 

- 

- 

- 

AddedSO 

Engages in interscholastic design competitions to motivate 
students . . . 

“ 

“ 

“ 

“ 

AddedSl 

. . .importance of nanotechnologies in 
developing. . .innovations 

“ 

“ 

“ 

“ 

Added52 

. . .convergence of nanoscience, biotech. . .information 
tech... cognitive sci 

“ 

“ 

“ 

“ 

Added53 

. . . knowledge of science and mathematics is critical to 
engineering. 

“ 

“ 

“ 

“ 

Added54 

there are many approaches to design and not just one 
“design process.” 

“ 

“ 

“ 

“ 


* Indicates consensus, t Mean and standard deviation are provided for reference only. Please note that 32 experts participated. 


Round 2 

In Round 2, for engineering outcomes for grades 9-12 for students who want to pursue 
engineering after graduation, participants were provided with their own ratings per item 
respectively from Round 1, were provided the ratings that fell within the IQR per item, were 
asked to rate items with the majority (within the IQR) or to justify why they did not join the 
majority, and provide comments. Only 32 of 34 Round-2 instruments were returned. The 
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rewordings and added items that were submitted from Round 1 were juried by the researchers 
and an engineer. An IQR of 1 or less represents consensus on an item’s rating. On the Round 2 
and Round 3 instruments the range of majority responses was always rounded out to the outer 
whole number rating. 

Thirty-one of 47 originally listed items achieved consensus after Round 2 as shown in 
Table 8. Three of seven new items (added by participants) achieved consensus after Round 2 for 
a total of 34 items in consensus. Each of these items’ median ratings was either 3 or 4. Therefore, 
no items could be dropped because of low median ratings. Thus the list grew instead of getting 
smaller. Items which remained at an IQR of 2 or more after Round 2 were, therefore, dropped 
from Round 3, because the researchers had received several complaints from participants about 


the length of the instrument, and the researchers wanted to maintain a good response rate. 


Table 8. Round 2 analysis compared to Round 1 analysis 




Round 2 

Round 1 

Item 

Outcome 

IQR 

Mdn 

Mn 

SD 

IQR 

Mdn 

Mn 

SD 

1-1- 

Is able to define engineering. 

2+ 

4 

4.03 

1.17 

2 

4 

4.03 

1.19 

2 

engineering future career 

1.75 

4.5 

4.21 

.946 

1.25 

4.5 

4.18 

.999 

3-t 

disciplines of engineering 

2.75 

3 

3.38 

1.26 

2.25 

3 

3.44 

1.24 

4* 

use, manage, assess technology. 

1* 

4 

3.88 

.808 

1.5 

4 

3.91 

.805 

5* 

Practices engineering ethics. 

1* 

3.5 

3.5 

1.11 

1.25 

4 

3.56 

1.16 

6 + 

Works effectively in teams 

2+ 

4 

3.85 

.989 

2 

4 

3.82 

1.03 

1 + 

Eng. design includes identifying needs. . . 

2+ 

4 

4.12 

.880 

1.25 

4 

4.18 

.869 

8* 

Uses models to study processes 

1* 

4 

3.53 

.825 

1* 

4 

3.59 

.988 

9 

design is iterative optimization 

1.75 

4.5 

4.24 

.890 

1.25 

5 

4.26 

.898 

10* 

Organizes design process. . . 

1* 

4 

3.56 

.705 

1* 

4 

3.62 

.853 

11* 

. . .economics. . .influence a solution. 

1* 

4 

3.74 

.864 

2 

4 

3.79 

.880 

12* 

. . . engineering principles . . . applied 

1* 

4 

3.62 

1.07 

1.25 

4 

3.65 

1.10 

13* 

. . . other factors . . . considered 

1* 

4 

3.65 

.691 

1* 

3.5 

3.68 

.843 

14* 

Uses optimization techniques 

1* 

3 

2.50 

.749 

1* 

3 

2.59 

.857 

15* 

Applies mathematics and science 

1* 

4 

4.26 

.790 

1* 

4 

4.26 

.898 

16* 

Uses a physical or math model 

1* 

3 

2.71 

.938 

2 

3 

2.94 

1.21 

17* 

. . .reverse engineering. . .can analyze 

1* 

3 

3.35 

.774 

1* 

3.5 

3.35 

.917 

18 

design includes. . . improvement. . . 

1.75 

4 

4.00 

.739 

2 

4 

3.94 

.814 

19* 

. . .creativity is. . .important 

1* 

4.5 

4.26 

.790 

1* 

4.5 

4.26 

.864 

20 

Applies research and development 

1.75 

3 

3.21 

1.01 

1.25 

3 

3.26 

1.14 

22* 

Designs, produces, tests prototypes 

1* 

4 

3.5 

.992 

1.25 

4 

3.59 

1.08 
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23 

... no perfect design. 

1.5 

4 

4.03 

.758 

2 

4 

3.85 

.989 

24* 

. . . human values when designing 

1* 

4 

3.68 

.727 

1* 

4 

3.74 

.864 

25 

solution to one problem create new prob. 

1.75 

4 

3.94 

.814 

2 

4 

3.97 

.883 

26+ 

Design. . .requires. . .constraints 

2+ 

4 

3.94 

.776 

1.25 

4 

3.85 

.857 

27+ 

Uses graphs to show relationships 

2+ 

4 

4.06 

.886 

1.25 

4 

4.12 

.913 

28* 

. . .personal computer operations 

1* 

4 

4.18 

.936 

1* 

4 

4.18 

.904 

29* 

. . .basic technical presentations 

1* 

4 

4.21 

.914 

1* 

4 

4.24 

.890 

30* 

engineering design portfolio. 

1* 

3 

3.15 

1.16 

2 

3 

3.18 

1.22 

31* 

Uses technical drawings 

1* 

4 

3.56 

.927 

1* 

4 

3.64 

1.03 

32 

computer-aided engineering. 

1.5 

3 

2.94 

.952 

2 

3 

3.00 

.985 

33* 

scale and proportion in design. 

1* 

3 

3.44 

.705 

1* 

3 

3.56 

.824 

34* 

Visualizes in three dimensions. 

1* 

4 

3.44 

.960 

1* 

4 

3.68 

.976 

35+ 

Uses technical sketching 

2+ 

3 

3.62 

1.02 

1.25 

3.5 

3.53 

1.09 

36* 

dimensioning and tolerancing. 

1* 

3 

2.68 

1.01 

1* 

3 

2.76 

1.08 

37* 

Uses computer-aided design 

1* 

3 

2.68 

.912 

1.25 

3 

2.71 

.970 

38* 

basic ergonomics 

1* 

3 

2.56 

.705 

1* 

3 

2.65 

.734 

39+ 

basic electronics concepts 

2+ 

3 

3.03 

.870 

2 

3 

3.06 

.983 

40* 

Uses measuring equipment 

1* 

4 

4.21 

.729 

1* 

4 

4.18 

.716 

41* 

use of tools for material processes. 

1* 

3 

3.35 

.774 

1* 

3 

3.32 

.843 

42* 

basic power and energy concepts. 

1* 

3.5 

3.59 

.957 

1.25 

3.5 

3.53 

1.08 

43* 

processes for manufacturing. . . 

0* 

3 

2.85 

.610 

.25* 

3 

2.97 

.797 

44* 

material processes 

1* 

3 

3.00 

.778 

1.25 

3 

3.03 

.797 

45* 

basic mechanics to engineering 

1* 

3 

3.29 

.719 

1* 

3 

3.35 

.774 

46* 

basic statics and strengths 

.75* 

3 

2.97 

.797 

2 

3 

3.09 

.933 

47* 

basic dynamics and motion 

1* 

3 

2.50 

.826 

1* 

3 

2.71 

1.06 

Added48* 

identify problems solved eng 

1* 

4.5 

4.31 

.850 

- 

- 

- 

- 

Added49 

Believes in his/her ability 

2 

4 

4.00 

1.07 

- 

- 

- 

- 

Added50 

interscholastic design competition 

2 

3 

2.96 

1.19 

- 

- 

- 

- 

Added51 

. . .importance of nanotechnologies 

2 

3 

2.86 

1.11 

- 

- 

- 

- 

Added52 

. . .convergence of nanoscience. . . 

2 

3 

2.64 

1.13 

- 

- 

- 

- 

Added53* 

science and mathematics is critical 

1* 

4 

4.46 

.508 

- 

- 

- 

- 

Added54* 

. . . are many approaches to design 

1* 

4 

3.52 

.975 

- 

- 

- 

- 


*Indicates consensus 

+Indicates that the item was dropped from Round 3 


There were numerous comments posted in the Round 1 and Round 2 instruments which 
reveal how some of the participating engineers think about these outcomes at the high school 
level. These comments were listed on each of the Round 2 and Round 3 instruments. 
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Round 3 

In Round 3, for engineering outcomes for grades 9-12 for students who want to pursue 
engineering after graduation, participants were provided with their own ratings per item 
respectively from Round 2, were provided the ratings that fell within the IQR per item and the 
median rating, were asked to rate items with the majority (within the IQR) or to justify why they 
did not join the majority, and provide comments. In order to keep the response rate high, it was 
decided to not ask participants to rank or order items. Going into Round 3, approximately 20 
items were rated at 3 and a similar number were rated at 4. There were no other ratings. 
Participants were not, therefore, asked to rank or order outcome items within a rating because it 
would be considered a hardship to ask them to rank 20 items in only two categories while still 
asking them to complete other tasks. 

Forty-three of the 54 total items achieved consensus after Round 3. Thus, Round 3 
provided participants with the opportunity to agree on nine additional items. Once again, ratings 
only consisted of 3 and 4. Twenty-one items were rated at 3 or Important to include in the 
curriculum and 21 items were rated at 4 or More Important to include in the curriculum. One 
item was rated at a 4.5 median, which may conceptually mean Most Important (mode=5). Table 


9 below shows a comparison of the first three rounds of the modified Delphi study. 


Table 9; A Comparison of the Analyses of the First Three Rounds 


Rounds 1, 2, & 3 Analyses Compared 

Round 3 

Round 2 

Round 1 

Item 

Outcome 

IQR 

Mdn 

Mn 

SD 

IQR 

Mdn 

Mn 

SD 

IQR 

Mdn 

Mn 

SD 

1 + 

Is able to define engineering. 

- 

- 

- 

- 

2 + 

4 

4.03 

1.17 

2 

4 

Mn 

SD 

2 - 

engineering future career 

1 - 

4 

4.09 

.963 

1.75 

4.5 

4.21 

.946 

1.25 

4.5 

4.03 

1.19 

3+ 

disciplines of engineering 

- 

- 

- 

- 

2.75 

3 

3.38 

1.26 

2.25 

3 

4.18 

.999 

44 s 

use, manage, assess technology. 

1 

4 

3.75 

.568 

1 * 

4 

3.88 

.808 

1.5 

4 

3.44 

1.24 

5* 

Practices engineering ethics. 

1 

3 

3.44 

.914 

1 * 

3.5 

3.5 

1.11 

1.25 

4 

3.91 

.805 

6+ 

Works effectively in teams 

- 

- 

- 

- 

2 + 

4 

3.85 

.989 

2 

4 

3.56 

1.16 

7+ 

engineering design includes... 

- 

- 

- 

- 

2 + 

4 

4.12 

.880 

1.25 

4 

3.82 

1.03 

8* 

Uses models to study processes 

1 

4 

3.50 

.718 

1 * 

4 

3.53 

.825 

1* 

4 

4.18 

.869 

9- 

design is iterative. ..optimization 

1 - 

4 

4.22 

.751 

1.75 

4.5 

4.24 

.890 

1.25 

5 

3.59 

.988 

10 * 

Organizes design process... 

1 

4 

3.56 

.564 

1 * 

4 

3.56 

.705 

1* 

4 

4.26 

.898 

11 * 

. . .economics. . .influence a solution. 

1 

4 

3.75 

.762 

1 * 

4 

3.74 

.864 

2 

4 

3.62 

.853 

12 * 

. . .engineering principles. . .applied 

1 

4 

3.53 

.950 

1 * 

4 

3.62 

1.07 

1.25 

4 

3.79 

.880 

13* 

. . . other factors. . . considered 

1 

4 

3.69 

.644 

1 * 

4 

3.65 

.691 

1* 

3.5 

3.65 

1.10 
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14* 

Uses optimization techniques 

1 

3 

2.53 

.621 

1* 

3 

2.50 

.749 

1* 

3 

3.68 

.843 

15* 

Applies mathematics and science 

1 

4 

4.28 

.581 

1* 

4 

4.26 

.790 

1* 

4 

2.59 

.857 

16* 

Uses a physical or math model 

1 

3 

2.53 

.718 

1* 

3 

2.71 

.938 

2 

3 

4.26 

.898 

17* 

. . .reverse engineering. . .can analyze 

1 

3 

3.34 

.787 

1* 

3 

3.35 

.774 

1* 

3.5 

2.94 

1.21 

18+ 

design includes. . . improvement. . . 

- 

- 

- 

- 

1.75 

4 

4.00 

.739 

2 

4 

3.35 

.917 

19* 

. . .creativity is. . .important 

1 

4 

4.41 

.615 

1* 

4.5 

4.26 

.790 

1* 

4.5 

3.94 

.814 

20- 

Applies research and development 

1- 

3 

3.28 

.729 

1.75 

3 

3.21 

1.01 

1.25 

3 

4.26 

.864 

22* 

Designs, produces, tests prototypes 

1 

4 

3.69 

.693 

1* 

4 

3.5 

.992 

1.25 

4 

3.26 

1.14 

23 

... no perfect design. 

0- 

4 

3.97 

.647 

1.5 

4 

4.03 

.758 

2 

4 

3.59 

1.08 

24* 

Takes human values when designing 

1 

4 

3.66 

.602 

1* 

4 

3.68 

.727 

1* 

4 

3.85 

.989 

25- 

solution to one problem create prob. 

.75- 

4 

3.97 

.695 

1.75 

4 

3.94 

.814 

2 

4 

3.74 

.864 

26+ 

Design... requires taking constraints 

- 

- 

- 

- 

2+ 

4 

3.94 

.776 

1.25 

4 

3.97 

.883 

27+ 

Uses graphs to show relationships 

- 

- 

- 

- 

2+ 

4 

4.06 

.886 

1.25 

4 

3.85 

.857 

28* 

. . .personal computer operations 

1 

4 

4.06 

.948 

1* 

4 

4.18 

.936 

1* 

4 

4.12 

.913 

29* 

. . .basic technical presentations 

1 

4 

4.16 

.808 

1* 

4 

4.21 

.914 

1* 

4 

4.18 

.904 

30* 

engineering design portfolio. 

1 

3 

3.09 

.734 

1* 

3 

3.15 

1.16 

2 

3 

4.24 

.890 

31* 

Uses technical drawings 

1 

4 

3.63 

.707 

1* 

4 

3.56 

.927 

1* 

4 

3.18 

1.22 

32- 

computer-aided engineering. 

0- 

3 

2.88 

.751 

1.5 

3 

2.94 

.952 

2 

3 

3.64 

1.03 

33* 

scale and proportion in design. 

1 

3 

3.47 

.507 

1* 

3 

3.44 

.705 

1* 

3 

3.00 

.985 

34* 

Visualizes in three dimensions. 

1 

3.5 

3.47 

.803 

1* 

4 

3.44 

.960 

1* 

4 

3.56 

.824 

35+ 

Uses technical sketching 

- 

- 

- 

- 

2+ 

3 

3.62 

1.02 

1.25 

3.5 

3.68 

.976 

36* 

dimensioning and tolerancing. 

1 

3 

2.66 

.865 

1* 

3 

2.68 

1.01 

1* 

3 

3.53 

1.09 

37* 

Uses computer-aided design 

1 

3 

2.72 

.813 

1* 

3 

2.68 

.912 

1.25 

3 

2.76 

1.08 

oo 

* 

basic ergonomics 

1 

3 

2.63 

.492 

1* 

3 

2.56 

.705 

1* 

3 

2.71 

.970 

39+ 

basic electronics concepts 

- 

- 

- 

- 

2+ 

3 

3.03 

.870 

2 

3 

2.65 

.734 

40* 

Uses measuring equipment 

1 

4 

4.19 

.592 

1* 

4 

4.21 

.729 

1* 

4 

3.06 

.983 

41* 

use of tools for material processes. 

1 

3 

3.25 

.622 

1* 

3 

3.35 

.774 

1* 

3 

4.18 

.716 

42* 

basic power and energy concepts. 

1 

3 

3.44 

.504 

1* 

3.5 

3.59 

.957 

1.25 

3.5 

3.32 

.843 

43* 

processes for manufacturing. . . 

0 

3 

2.84 

.448 

0* 

3 

2.85 

.610 

.25* 

3 

3.53 

1.08 

44* 

material processes 

0 

3 

2.97 

.695 

1* 

3 

3.00 

.778 

1.25 

3 

2.97 

.797 

45* 

basic mechanics to engineering 

1 

3 

3.28 

.457 

1* 

3 

3.29 

.719 

1* 

3 

3.03 

.797 

46* 

basic statics and strengths 

.75 

3 

2.78 

.608 

.75* 

3 

2.97 

.797 

2 

3 

3.35 

.774 

47* 

basic dynamics and motion 

1 

3 

2.56 

.669 

1* 

3 

2.50 

.826 

1* 

3 

3.09 

.933 

48* 

identify problems solved eng 

1 

4.5 

4.47 

.567 

1* 

4.5 

4.31 

.850 

- 

- 

2.71 

1.06 

49- 

Believes in his/her ability 

1- 

4 

4.19 

.792 

2 

4 

4.00 

1.07 

- 

- 

- 

- 

50 

interscholastic design competitions 

2 

3 

2.97 

1.05 

2 

3 

2.96 

1.19 

- 

- 

- 

- 

51- 

...importance of nanotechnologies 

1- 

3 

2.69 

.965 

2 

3 

2.86 

1.11 

- 

- 

- 

- 

52- 

...convergence of nanoscience, bio 

1- 

3 

2.59 

.911 

2 

3 

2.64 

1.13 

- 

- 

- 

- 

53* 

science and mathematics is critical 

1 

4 

4.41 

.499 

1* 

4 

4.46 

.508 

- 

- 

- 

- 

54* 

there are many approaches to design 

0 

4 

3.88 

.660 

1* 

4 

3.52 

.975 

- 

- 

- 

- 


*Indicates consensus 

+Indicates that the items was dropped from Round 3 because the item's IQR was still 2 or more after Round 2. 
- Indicates that consensus was reached in Round 3 


Rounds 4, 5, and 6, Engineering OutcomeGroup Rankings 

Because it would be difficult to rank outcome items into order of importance within each 
of the only two rating groups (Important and More Important), the researchers decided to have 
selected engineers group outcome items into groups of conceptual alikeness and name the 
groupings with a category name. This would prepare the Round 4 instrument for the modified 


Delphi participants to rank each category only. The same basic statistic for consensus, an IQR of 
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1, was used for Rounds 4, 5, and 6. Only 19 of the original 32 agreed to participate in these 
additional last three rounds of the study. After these last three rounds (rounds 4, 5, and 6) 
dedicated to ranking the groupings of outcomes, the participants could only agree on what should 
be taught l'^^, and 7*. The final engineering outcome grouping names and their outcome 
group rankings are presented in Table 10. 

The following grouping summaries characterize each grouping of engineering outcomes. 

Engineering design. This grouping of outcomes emphasizes the big picture when it 
comes to engineering design. It emphasizes the importance of creativity and confidence when it 
comes to designing engineered solutions to problems. There was also consensus within this 
grouping as to the importance of outcomes related to design iteration, varying design processes, 
and tradeoffs. 

Application of engineering design. This grouping includes outcomes related to specific 
design activities. For example, students should be able to organize and optimize the overall 
engineering design process. Experimentation, prototyping, and reverse engineering are included 
in this grouping. 

Engineering analysis. In this grouping of outcomes, mathematics is emphasized. This is 
the grouping that includes using mathematics to optimize solutions, and it emphasizes the use of 
mathematics and science in the engineering design process. 

Engineering and human values. This grouping of outcomes emphasizes the big picture 
when it comes to the interaction of engineering design and society. It includes, for example, the 
weighing of limitations with decisions about safety and the environment versus costs and ethics. 

Engineering communication. This grouping includes a variety of outcomes ranging 
from CAD to presenting solutions in a variety of formats such as graphical, verbal, and 
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numerical. The group tends to characterize all sorts of communications important to the 
engineering design process. 

Engineering science. This grouping includes many of the traditional engineering 
sciences such as statics and dynamics. It includes items like understanding material properties 
and materials processes, ergonomics, energy and power, et cetera. 

Emerging fields of engineering. This grouping of outcomes includes two items related 
to nanotechnology, but it is understood as being able to include such fields as genetic 
engineering, biotechnology, and smart materials to name just a few of the possibilities. 

Table 10: Ranking of the Outcome Items within Categories; Results from Round 6 


Rating 

Rank 

Outcome Group and Outcome Consensus Items 

from 
Rounds 
1, 2, 3 

from Round 
6 


IQR = 0 
Mode =1.0 
Median =1.0 
Mean =1.5 
*SD= 1.30 

Ensineerine Design 

Regarding engineering outcomes related to Engineering Design the student in grades 9 through 
12: 

4 

Rank 

ist 

Understands that engineering design is an iterative process. 

4 

Is aware of how engineering principles must be applied when designing engineering solutions to 
problems. 

4 

Understands that creativity is an important characteristic for engineers to apply in design. 

4 

Believes in his/her ability to design a solution to a problem. 

4 

Recognizes that there are many approaches to design and not just one “design process.” 

4 

Understands engineering as it is actually practiced as a future career option. 


1QR = 2 
Mode = 2.0 
Median = 3.0 
Mean = 3.0 
*SD= 1.15 

Annlication of Ensineerine Design 

Regarding engineering outcomes related to Application of Engineering Design the student in 
grades 9 through 12: 

4.5 

Rank 

undetermined 

Is able to identify problems that could be solved through engineering design. 

4 

Organizes and manages the engineering design process that includes optimal use of materials, 
processes, time, and expertise. 

4 

Designs, produces, and tests prototypes of products. 

4 

Understands that there is no perfect design. Designs that are best in one respect may be inferior in 
other ways (cost or appearance). Usually some features must be sacrificed as trade-offs to gain other 
features. 

3 

Conducts reverse engineering and can analyze how a product or process was designed and created. 

3 

Applies research and development and experimentation in the production of new or improved 
products, processes, and materials. 


1QR= 1 
Mode = 3.0 

Engineering Analysis 
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Median = 3.0 
Mean = 3.4 
*SD = .768 

Regarding engineering outcomes related to Engineering Analysis the student in grades 9 
through 12: 

4 

Rank 

3 rd 

Uses models to study processes that cannot be studied directly. 

4 

Applies mathematics and science to the engineering process. 

4 

Uses measuring equipment to gather data for troubleshooting, experimentation, and analysis. 

4 

Understands that knowledge of science and mathematics is critical to engineering. 

3 

Uses a physical or mathematical model to estimate the probability of events. 

3 

Uses optimization techniques to determine optimum solutions to problems. 


IQR = 3 
Mode = 5.0 
Median = 5.0 
Mean = 4.3 
*SD= 1.64 

Ensineerine and Human Values 

Regarding engineering outcomes related to Engineering and Human Values the student in 
grades 9 through 12: 

3 

Rank 

undetermined 

Practices engineering ethics. 

4 

Is aware of how societal interests, economics, ergonomics, and environmental considerations 
influence a solution. 

4 

Understands how other factors, such as cost, safety, appearance, environmental impact, and what 
will happen if the solution fails must be considered when designing engineering solutions to 
problems. 

4 

Takes human values and limitations into account when designing and solving problems. 

4 

Understands that the solution to one problem may create other problems. 



Comment: Understands that engineers have societal obligations and responsibilities. 
{Temporarily added by juror to provide panel with a better characterization of this 
grouping of outcomes.) 


1QR = 3 
Mode = 6.0 
Median = 4.0 
Mean = 4.3 
*SD= 1.37 

Eneineerine Communication 

Regarding engineering outcomes related to Engineering Communication the student in grades 
9 through 12: 

4 

Rank 

undetermined 

Understands basic personal computer operations and uses basic computer applications such as word 
processors, spreadsheets, and presentation software. 

4 

Provides basic technical presentations, graphics, and reports, and communicates verbally information 
related to engineering processes. 

4 

Uses technical drawings to construct or implement an object, structure, or process. 

3.5 

Visualizes in three dimensions. 

3 

Develops and maintains an engineering design portfolio. 

3 

Understands computer-aided engineering. 

3 

Understands scale and proportion in design. 

3 

Applies the rules of dimensioning and tolerancing. 

3 

Uses computer-aided design to construct technical drawings. 


1QR = 3 

Mode = 5.0 and 6.0 
Median = 5.0 
Mean = 4.4 
*SD= 1.67 

Eneineerine Science 

Regarding engineering outcomes related to Engineering Science the student in grades 9 
through 12: 

4 

Rank 

undetermined 

Understands engineering as it is actually practiced as a future career option. 

4 

Develops basic ability to use, manage, and assess technology. 

3 

Applies knowledge of basic ergonomics to the engineering process. 

3 

Develops basic skill in the use of tools for material processes. 

3 

Applies basic power and energy concepts. 

3 

Applies knowledge of the processes for manufacturing products to the engineering process. 

3 

Applies knowledge of material processes to the engineering process. 

3 

Applies knowledge of basic mechanics to the engineering process. 

3 

Applies knowledge of basic statics and strengths of materials to the engineering process. 

3 

Applies knowledge of basic dynamics and motion of rigid bodies and particles to the engineering 
process. 
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IQR = 0 
Mode = 7.0 
Median = 7.0 
Mean = 6.8 
*SD = .315 

Emereins Fields of Eneineerine 

Regarding engineering outcomes related to Emerging Fields of Engineering the student in 
grades 9 through 12: 

3 

Rank 

^th 

Understands the importance of nanotechnologies in developing the next generation of innovations 
(less power, smaller). 

3 

Understands the convergence of nanoscience, biotechnology, information technology and how 
cognitive science creates opportunities for the improvement of industrial productivity and quality of 
human life. 


Comment: Understands that engineering is a set of living and evolving fields from which 
new technologies and concepts emerge constantly. (Temporarily added by juror to provide 
panel with a better characterization of this grouping of outcomes.) 


*The mean and standard deviation are included for reference only. Please note that only 19 participants were involved with the 


grouping extension of the study (rounds 4, 5, and 6). 


Discussion 

It is an important finding that participants could not agree on an outcome that would likely 
be considered important by the NCETE, pre engineering teachers, and other educators. Item 
seven still had an IQR of 2 after Round 2. The wording of the item follows below. 


Regarding engineering outcomes related to Engineering Design the student in grades 9 
through 12: 

Item 7 : 

Understands that engineering design involves identifying needs for technical solutions, 
using human information resources to obtain ideas, considering constraints, generating 
alternative solutions, developing drawings with measurements and details of construction, 
constructing models, testing the solution against design specifications, and suggesting 
modifications for improvement. 

However, in Round 2, the following item, which was added by the participants in Round 1, 
gained consensus. 


Regarding engineering outcomes related to Engineering Design the student in grades 9 
through 12: 

Item 54: IQR 1, Mdn 4 

Recognizes that there are many approaches to design and not just one “design process.” 
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It is plausible that one reason that consensus could not be formed regarding Item 7 above is 
that it was worded so long and had so many individual components. One indicator that lends 
support to this theory is that a participant commented, “This item is too complex to rate fairly. I 
have different reactions to different parts of it.” Another indicator of this plausibility is that the 
individual components that make up Item 7 appear individually as separate items which did gain 
consensus. Those items are shown below. 


Item 48: IQR 1, Mdn 4.5 (If the median does not become a whole number after Round 3, 
the mode will be used to characterize the rating.) 

Is able to identify problems that could be solved through engineering design. 

Item 8: IQR 1, Mdn 4 

Uses models to study processes that cannot be studied directly. 

Item 11: IQR l,Mdn4 

Understands how societal interests, economics, ergonomics, and environmental 
considerations influence a solution. 

Item 13: IQR 1, Mdn 4 

Understands how other factors, such as cost, safety, appearance, environmental impact, and 
what will happen if the solution fails must be considered when designing engineering 
solutions to problems. 

Item 14: IQR 1, Mdn 3 

Uses optimization techniques to determine optimum solutions to problems. 

Item 22: IQR 1, Mdn 4 

Designs, produces, and tests prototypes of products. 

Item 24: IQR 1, Mdn 4 

Takes human values and limitations into account when designing and solving problems. 
Item 37: IQR 1, Mdn 3 

Uses computer-aided design to construct technical drawings. 

Item 29: IQR 1, Mdn 4 

Provides basic technical presentations, graphics, and reports, and communicates verbally 
information related to engineering processes. 
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Wicklein’s (2006) premise that the use of mathematics and science in order to optimize 
solutions prior to implementation, for modeling and predictive analysis, and to generally support 
the engineering design process tends to be validated by the findings. However, while the NCETE 
tends to place a great deal of importance on optimization and prediction because those tend to be 
missing in practice in technology education programs, the participants found those outcomes to 
be necessary or important but not essential or more important. Some comments were posted that 
these processes below were beyond the abilities of high school students. 

IQR l,Mdn4 

Applies mathematics and science to the engineering process. 

IQR 1 Mdn 3 

Uses optimization techniques to determine optimum solutions. . . 

IQR I Mdn 3 

Uses a physical or mathematical model to estimate. . .probability of events. 

Additional items of interest about which many of the NCETE partners and other educators 
may be curious, regarding the design of NCETE professional development activities, include the 
following items that gained consensus: Items 38 and 40 through 47 (refer to the Table 9 above). 

It is interesting that consensus items had medians of either 3 (meaning the item is necessary 
or important) or 4 (meaning the item is essential or more important). It is plausible that this 
finding is due to the fact that those standards published by the resources cited above are valid in 
terms of engineering outcomes. Eurthermore, the narrow range of ratings for consensus items 
means that the NCETE and other educators can use those consensus outcomes with a fair level of 
confidence regarding their validity. 

Of further interest is that so many items tend to support the conclusions of Robinson, 
Sparrow, Clegg, and Birdi and the NAE regarding the competencies and attributes of future 
engineers. Eor example. Item 19 (IQR 1, Mdn 4) emphasizes the NAE's conclusion that 
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creativity is a key engineering attribute. It states, "Understands that creativity is an important 
characteristic for engineers to apply in design." Regarding the NAE's conclusion that flexibility 
will be a more important attribute, it is interesting that participants added and reached consensus 
on Item 54 (IQR 0, Mdn 4), "Recognizes that there are many approaches to design and not just 
one design process.” Participants, like the NAE, may recognize that flexibility will be needed in 
solving a wide variety of problems through engineering, and this may also be based on their 
experiences. As a matter of efficiently managing complexity, both Robinson, Sparrow, Clegg, 
and Birdi and the NAE conclude that the engineer's ability to organize the engineering process 
will be even more important in the future. Item 10 directly addresses that concern. Item 10 states, 
"Organizes and manages the engineering design process that includes optimal use of materials, 
processes, time, and expertise." Both Robinson, Sparrow, Clegg, and Birdi and the NAE 
emphasize that future engineers will have to understand the various influences on designs and 
design tradeoffs and practice ethics, and it is interesting to note that Items 5, 11, and 13 reflect 
those same concerns. They are listed below. 

Item 5, IQR 1, Mdn3: Practices engineering ethics. 

Item 11, IQR 1, Mdn 4: Understands how societal interests, economics, ergonomics, and 
environmental considerations influence a solution. 

Item 13, IQR 1, Mdn 4: Understands how other factors, such as cost, safety, appearance, 

environmental impact, and what will happen if the solution fails must be 
considered when designing engineering solutions to problems. 

That both Robinson, Sparrow, Clegg, and Birdi and the NAE conclude that engineers will 
need to have broader foundations of knowledge regarding emerging or revolutionary 
technologies, to the extent that an extra year or two may need to be added to traditional 
undergraduate engineering education, it is noteworthy that nanotechnology was included as 
Important in both Items 51 and 52 each with IQRs of 1 and medians of 3. These items were 
added by participants. No other emerging technologies such as biotechnology were identified by 
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participants. The addition of nanotechnology may suggest that there is concern that students 
understand emerging technologies, and perhaps that concern has not yet peaked among 
engineers. 

It is also interesting to note from a technology education point of view, that the 
participants could not reach consensus regarding the necessity of including technical sketching 
but did find that CAD is necessary. This somewhat contradicts the findings of the Bearing and 
Daugherty study. However, that study did included technology educators in addition to 
engineering educators, and it is plausible that technology educators place more importance on 
sketching than do engineers. When it came to making models and prototypes for testing and 
analysis, participants found that this was essential with a median of 4, however, some 
participants commented that “this sounds suspiciously like shop class” and suggested on more 
than one occasion that such hands-on activities would be a turn off to students. It is not clear 
whether such a perspective is contrary to guidelines developed by Douglas, Iverson, and 
Kavandurg (2004), which call for engineering education at the K-12 level to be a hands-on 
learning experience. After all, it is quite possible to have hands-on learning experiences without 
actually making an authentic prototype. 

Additionally, both Robinson, Sparrow, Clegg, and Birdi and the NAE conclude that 
engineers will need to work in teams, including teams that include non-engineers. However, the 
participating engineers and engineering educators did not reach consensus on the study's related 
item, "Works effectively in teams." There were comments written by participants questioning the 
need for students to work in groups. Also noteworthy is the lack of consensus on Items I and 3. 
They respectively read, "Is able to define engineering," and "Understands the disciplines of 
engineering." Comments made by participants regarding these items allude to the trivial nature 
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of such outcome items and that more emphasis should be placed on outcomes that make students 
want to be engineers. 

Finally, the fact that the participants were only able to reach consensus on the rankings of 
three of the outcomes groupings appears to be explained by fundamental disagreement as to 
which groupings of outcomes should be taught first, second, et cetera. Like in the first three 
rounds of the study, participants had to post comments if they did not vote with the majority. 
These comments indicated a sustained disagreement. Nevertheless, with IQR’s of 0 (zero) it is 
clear that participants were able to agree that Engineering Design should be ranked first in 
importance, or the most important to get taught in a limited time frame and that Emerging Eields 
of Engineering was last in importance, or the least important to get taught in a limited time 
frame. 

Some researchers who have seen the results of this study prior to publication were 
surprised that the outcomes that reached consensus were not more “global” such as those 
promoted by the NAE committee that provided input for the conclusions reached in The 
Engineer of 2020. Two of these researchers have suggested that the participants should have only 
included engineering professors who teach freshmen level engineering courses at the college 
level. However, the researchers of this study were advised to seek nominations by the NAE and 
ABET. Recommendations from other researchers in the NCETE, ABET, and the NAE focused 
on including collegiate engineering educators who are familiar with K-12 education as much as 
possible and to include engineering professors and practicing engineers as much as possible for 
balance. Nevertheless, having a homogeneous group such as, only freshmen level engineering 
design professors, would be an excellent approach for future studies that are similar to this one. 



48 


Regarding the usefulness of the outcomes study, the reader should understand that Delphi 
studies use relatively small participant sizes because the process is dependent upon the 
participants being experts in their fields. It organizes expert opinion. Therefore, one should not 
be reluctant to consider these findings as input to curriculum decisions. It is interesting that 
consensus items had medians of either 3 (meaning the item is Important or necessary) or 4 
(meaning the item is More Important or essential). It is plausible that this finding is due to the 
fact that those standards published by the resources cited above are valid in terms of engineering 
outcomes. Furthermore, the narrow range of ratings for consensus items means that the NCETE 
and others can use those consensus outcomes with a good level of confidence. However, were 
the study to be repeated, the researchers should consider constraining participants to the number 
of outcomes that can hold a particular rating. Eor example, only one-fifth of the outcomes can be 
rated at 1, Eeast Important, and so on. Euture researchers should also consider expanding the 
rating scale from a five-point scale to a 10-point scale. Certainly, the Delphi process used for this 
study was influenced by "regression toward the mean" as indicated by the fact that only one 
consensus item achieved a mode of 5 as its rating. No consensus items achieved ratings of 1 or 2. 
Nevertheless, participants had the opportunity to rate items, and there was not consensus 
regarding any item being rated at the 1 or 2 level. Moreover, the interquartile range was 
deliberately used to narrow the influence of out- lying data on the determination of consensus, 
which also provides an additional level of confidence in the use of these findings in high school 
engineering curricula. To date, no correlations among demographic variables and outcome 


ratings have been run. 
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Recommendations 

The following recommendations will be of interest to teacher educators, teachers of 
technology education, teachers of pre engineering, engineering educators, administrators, and the 
NCETE. 

1. Have a person with influence and stature (who can convince engineers to participate in 
focus groups) to lead focus groups of prominent engineers for more insight on 
engineering outcomes and issues related to teaching engineering concepts in grades 9-12. 
Such a person may also be able to convince engineers to participate in a Delphi study that 
does not start with pre listed items. 

2. One advantage of conducting a Delphi study is that people who may have outstanding 
stature or who may tend to dominate discussions, have less biasing influence on the 
consensus-building process. Nevertheless, it may well be that some decisions are best 
made in face-to-face meetings. Therefore, it is recommended that a workshop be 
conducted on engineering outcomes, in which experts have a chance to more deliberately 
persuade one another about the importance of outcomes and groupings of outcomes. 

3. Conduct a replication study. 

4. Enhance technology education by infusing selected engineering outcomes into the 
technology education curriculum for non-pre engineering curricula. The researchers find 
less utility in making engineering the focus of technology education programs which 
focus on general technological literacy but believe that adding selected outcomes is 
useful. Therefore, they recommend conducting a similar study in which technology 
education supervisors, teachers, and teacher educators identify those engineering 
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consensus outcomes identified herein for inclusion in technology education programs 
which focus on technological literacy. 

5. Use these outcomes to aid in the design of pre engineering programs. 

6. Use these outcomes to review existing pre engineering programs. 

7. Use these outcomes as contexts to make mathematics and science more practical and 
motivating. 

8. Use these findings to redesign NCETE professional development. 

9. Upon findings in the technology education study recommended above, recommend a 
listing of engineering outcomes that can be infused into technology education programs 
for the purpose of providing technological literacy. 

10. Conduct a similar study in which the panel of experts is comprised only of engineering 
educators who teach freshmen engineering students at the college level. 

Implications for Technology Education Curriculum and Instruction 

Some of the implications of the core engineering outcomes identified herein are evident. 
Eor example, it is clear that engineering education at the K-12 level should be hands-on 
(Douglas, Iverson, & Kavandurg, 2004). So it would be necessary to include outcomes such as 
those related to conducting reverse engineering, research and development, and the fabrication of 
prototypes. It also seems fairly obvious that any program would include a breadth of engineering 
communication activities related to presenting findings, to using CAD, to using the computer as 
a means to control data and communicate engineering processes. Any program that taught 
engineering would benefit from having students apply mathematics and science principles to the 
solutions that they design. In the midst of an extended back-to-basics movement with high-stakes 
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testing, being able to improve student aehievement in, and attitudes toward STEM subjeets 
would provide a meaningful service to education and, perhaps cause an increase in the diversity 
of those students who would like to pursue STEM related careers after high school and college. 

What engineering outcomes should be included in a high school technology education 
program that focuses on providing students with technological literacy? Certainly, those 
outcomes that most closely correspond to the Standards for Technological Literacy, such as 
optimization, the realization that there are many societal factors that influence engineered 
solutions, and any outcome that will help students become better designers and understand the 
essence of what engineering is in real life, such as prototyping, creativity, and clearly managing 
the design process. Research and development and analysis are also important. 

What engineering outcomes should be included in a high school technology education 
program that focuses on pre engineering? All of those consensus outcomes from the Childress 
and Rhodes study were identified on the premise that they were to be taught to high school 
students who want to pursue engineering after they graduate. However, the curriculum designer 
should be careful. A crowded curriculum, which leaves no time for application, diminishes its 
effect on student achievement and motivation. Some outcomes need to be taught and applied 
repeatedly across the school year. These fundamental processes are the essence of engineering. 
Other outcomes need only be taught and applied once within a specific course. Perhaps the most 
pertinent approach to deciding what outcomes to include in a pre-engineering curriculum is 
building a course sequence that includes the outcomes in order of importance but also in order of 
prerequisites. Couple with that sequence, estimates of time to deliver instruction in a hands-on 
and motivating way. Where necessary, pre-engineering programs that use these outcomes should 
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consider dividing content so it is studied over a sequence of courses over a sequence of grade 
levels, while avoiding too many prerequisite courses that will limit enrollment. 

Having identified those core engineering concepts that should be taught to high school 
students, under what circumstances should one go about teaching the concepts? Douglas, 

Iverson, and Kavandurg (2004) in summarizing the results of an ASEE analysis of current 
practices in K-12 engineering education, developed the following guidelines for the future of K- 
12 engineering education. One, engineering education should be hands-on in order to motivate 
students by couching engineering problems in interesting and relevant social contexts. Two, 
engineering education should be taught in an interdisciplinary approach in order to show the 
relevancy of mathematics, science, and other subjects, by making engineering a conceptual place 
for the application of these subjects. Three, develop K-12 standards for use in lesson plans that 
help teachers teach mathematics and science concepts in the classroom. Douglas, Iverson, and 
Kavandurg suggest that state-developed K-12 standards should be developed like Massachusetts 
has published. Eour, improve teachers by providing more pay, more professional development, 
and more curriculum writing. Eive, make engineering a more attractive career choice for girls 
and minorities by working with their schools through outreach efforts. Six, engage more 
constituents in partnerships that cross all levels of the educational process. 

Teachers, teacher educators, and administrators, carry a heavy burden. Their mission is to 
provide students with the best education possible. Traditional education, memorizing facts, and 
studying textbooks has worked to some extent for some professions. However, the world has 
reached a crossroads; a point at which the modus operandi no longer works in terms of fostering 
inclusiveness. The engineering profession and all STEM subjects can attract a more diverse 
population of participants by providing access. This is not simply access to school and books but 
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access to instruction that has meaning for students who have not traditionally pursued STEM 
professions. Teaching meaningful concepts and providing meaningful opportunities for 
application is part of what "access" to education is about. 
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